Deterministic Simulation Testing

Tau DST is split into two crates:

• libdst — generic framework: DualSimulation, behavior tree, deterministic scheduler, shrink, fault helpers. Usable for any target + isolated reference model.
• dst — Tau driver binary. Implements the framework for every libtau::Executor storage configuration.

CLI

User-facing flags (plus --help):

Output is tracing only (RUST_LOG=warn recommended).

# Local run (all profiles + optional concurrent)
cargo run --release --bin dst -- --seed 42 --ops 2000 --concurrency 4

# CI (single command: all backends + concurrent)
RUST_LOG=warn cargo run --release --bin dst -- --ci --seed 1

Architecture

DualSimulation

Every run is a DualSimulation: pick an op, apply it to both target and model, compare outputs. The framework lives in libdst:

pick(rng) -> Op
apply(step, op) -> Vec<Divergence>   // structured mismatches
checkpoint(step, n, log, rng) -> CheckpointAction

Divergence records the step index, a description string, and Debug-formatted expected vs got values. CheckpointAction is either Continue { divergences } (keep log) or ResetLog { divergences } (discard log after WAL truncation).

Independent oracle

The reference oracle (crates/dst/src/oracle.rs) shares no code with libtau. It stores Vec<TauInterval> per layer and runs its own sweep-line compaction at the same threshold as the SUT. Divergences in libtau's sweep-line or query paths are caught because the oracle computes the same results independently.

Behavior tree

A static LazyLock<Tree<SimCtx, Op>> of 20 closure-based leaves. Guards and builders are Arc<dyn Fn> — no fn-pointer constraints. Tag bits suppress WAL-excluded ops at runtime (excluded_tags parameter to Tree::pick).

Deterministic scheduler

libdst::Scheduler implements cooperative concurrency without OS threads. A seeded RNG picks which task runs next. Every interleaving is reproducible from the seed. Use it to simulate multi-client concurrent workloads in integration tests.

Shrink

libdst::shrink and shrink_with_granularity reduce a failing op trace to the smallest sub-sequence that still fails, using the delta-debugging algorithm. Useful when a divergence is found after hundreds of ops and you need to understand the minimal reproducer.

Profile matrix

Profiles are a Cartesian product in profile/spec.rs: storage × compaction × encryption × transport × auth. Names look like wal_stress_enc_single_direct_noauth.

Every sequential run uses a fresh isolated oracle (never seeded from the executor). The target is either a direct Executor or a TCP/TLS WireClient talking to an ephemeral tau harness.

RUST_LOG=warn cargo run --release --bin dst -- --seed 42 --tier nightly

Each profile in a run is driven with the same --seed (re-seeded per profile for reproducibility).

Faults (checkpoint every 200 ops)

Damage kinds — a short write (truncate) vs a length-preserving bit-flip run (corrupt) — are drawn from the seeded RNG so both fire across the matrix. File faults reopen-probe the damaged store and assert tau recovers or returns a clean error, never panics; the store is then rebuilt from the authoritative op log, so the damage never perturbs the oracle comparison.

The disk corruption probe caught a real bug: a corrupted .dat could decode an inverted [start, end) interval and panic in Tau::new. The loader now validates intervals and bounds untrusted length prefixes, returning InvalidData instead.

Transactions are enabled for memory/disk/wire profiles. WAL profiles use WAL_EXCLUDED tags in the behavior tree to skip transaction and multi-DB ops until single-DB WAL replay semantics are fully validated.

For TTL, DST pins the wall clock via wall_clock::set_fixed_now_secs (1_700_000_000).

The disk backend (when selected) pairs each <db>.dat with a <db>.wal: appends and schema DDL (CREATE/DERIVE/SET TTL/DROP) go to the WAL first (fsynced by default), and <db>.dat is rewritten atomically only on checkpoint (compaction or [wal].max_size_mb). This makes acknowledged DML and lens definitions durable across clean restarts; CREATE DATABASE <name> on a disk executor re-opens <db>.dat, replays <db>.wal on top, and replays the schema section to restore base/derived lenses and TTL policies. The pbt_disk_persists_data_and_schema_across_reopen test in sim.rs covers this path under the DST harness.

Concurrent phase

When --concurrency > 0 (or --ci with default 4 readers), an in-memory writer/readers phase runs after all sequential profiles. Writes use the same dual-apply path; readers check for invalid RANGE shape (non-overlapping, sorted), then reconcile all AT values against the oracle after writes complete.

CI

After cargo nextest run --release, the workflow runs:

cargo test -p libdst -p dst --release
RUST_LOG=warn cargo run --release --bin dst -- --ci --seed 1

Then the Docker image build.

Tests

cargo nextest run --release -p libdst   # framework: btree, divergence, scheduler, shrink, ...
cargo nextest run --release -p dst      # driver: oracle, apply, btree, sim profiles

All #[hegel::test] property-based tests across crates are named with a pbt_ prefix (e.g. pbt_...) so they are easy to filter in logs and CI output.

See Testing for the full strategy.